User requirements for multimedia indexing and retrieval of unedited audio-visual footage - RUSHES

Multimedia analysis and reuse of raw un-edited audio visual content known as rushes is gaining acceptance by a large number of research labs and companies. A set of research projects are considering multimedia indexing, annotation, search and retrieval in the context of European funded research, but only the FP6 project RUSHES is focusing on automatic semantic annotation, indexing and retrieval of raw and un-edited audio-visual content. Even professional content creators and providers as well as home-users are dealing with this type of content and therefore novel technologies for semantic search and retrieval are required. As a first result of this project, the user requirements and possible user-scenarios are presented in this paper. These results lay down the foundation for the research and development of a multimedia search engine particularly dedicated to the specific needs of the users and the content

ELAN as flexible annotation framework for sound and image processing detectors

Annotation of digital recordings in humanities research still is, to a largeextend, a process that is performed manually. This paper describes the firstpattern recognition based software components developed in the AVATecH projectand their integration in the annotation tool ELAN. AVATecH (AdvancingVideo/Audio Technology in Humanities Research) is a project that involves twoMax Planck Institutes (Max Planck Institute for Psycholinguistics, Nijmegen,Max Planck Institute for Social Anthropology, Halle) and two FraunhoferInstitutes (Fraunhofer-Institut für Intelligente Analyse- undInformationssysteme IAIS, Sankt Augustin, Fraunhofer Heinrich-Hertz-Institute,Berlin) and that aims to develop and implement audio and video technology forsemi-automatic annotation of heterogeneous media collections as they occur inmultimedia based research. The highly diverse nature of the digital recordingsstored in the archives of both Max Planck Institutes, poses a huge challenge tomost of the existing pattern recognition solutions and is a motivation to makesuch technology available to researchers in the humanities

Application of video processing methods for linguistic research

Evolution and changes of all modern languages is a well-known fact. However, recently it is reaching dynamics never seen before, which results in loss of the vast amount of information encoded in every language. In order to preserve such heritage, properly annotated recordings of world languages are necessary. Since creating those annotations is a very laborious task, reaching times 100 longer than the length of the annotated media, innovative video processing algorithms are needed, in order to improve the efficiency and quality of annotation process

AVATecH ― automated annotation through audio and video analysis

In different fields of the humanities annotations of multimodal resources are a necessary component of the research workflow. Examples include linguistics, psychology, anthropology, etc. However, creation of those annotations is a very laborious task, which can take 50 to 100 times the length of the annotated media, or more. This can be significantly improved by applying innovative audio and video processing algorithms, which analyze the recordings and provide automated annotations. This is the aim of the AVATecH project, which is a collaboration of the Max Planck Institute for Psycholinguistics (MPI) and the Fraunhofer institutes HHI and IAIS. In this paper we present a set of results of automated annotation together with an evaluation of their quality

Application of audio and video processing methods for language research

Annotations of media recordings are the grounds for linguistic research. Since creating those annotations is a very laborious task, reaching 100 times longer than the length of the annotated media, innovative audio and video processing algorithms are needed, in order to improve the efficiency and quality of annotation process. The AVATecH project, started by the Max-Planck Institute for Psycholinguistics (MPI) and the Fraunhofer institutes HHI and IAIS, aims at significantly speeding up the process of creating annotations of audio-visual data for humanities research. In order for this to be achieved a range of state-of-the-art audio and video pattern recognition algorithms have been developed and integrated into widely used ELAN annotation tool. To address the problem of heterogeneous annotation tasks and recordings we provide modular components extended by adaptation and feedback mechanisms to achieve competitive annotation quality within significantly less annotation time

Effects of body size, sex, parental care and moult strategies on auk diving behaviour outside the breeding season

Information on seabird foraging behaviour outside the breeding season is currently limited. This knowledge gap is critical as this period is energetically demanding due to post‐fledging parental care, feather moult and changing environmental conditions. Based on species’ body size, post‐fledging parental strategy and primary moult schedule we tested predictions for key aspects of foraging behaviour (Maximum Dive Depth (MDD), Daily Time Submerged (DTS) and Diurnal Dive Activity (DDA)) using dive depth data collected from three seabird species (common guillemot Uria aalge, razorbill Alca torda and Atlantic puffin Fratercula arctica) from the end of the breeding season (July) to mid‐winter (January). We found partial support for predictions associated with body size; guillemots had greater MDD than razorbills but MDD did not differ between razorbills and puffins, despite the former being 35% heavier. In accordance with sexual monomorphism in all three species, MDD did not differ overall between the sexes. However, in guillemots and razorbills there were sex‐specific differences, such that male guillemots made deeper dives than females, and males of both species had higher DTS. In contrast, there were no marked sex differences in dive behaviour of puffins in July and August in accordance with their lack of post‐fledging parental care and variable moult schedule. We found support for the prediction that diving effort would be greater in mid‐winter compared to the period after the breeding season. Despite reduced daylight in mid‐winter, this increase in DTS occurred predominantly during the day and only guillemots appeared to dive nocturnally to any great extent. In comparison to diving behaviour of these species recorded during the breeding season, MDD was shallower and DTS was greater during the non‐breeding period. Such differences in diving behaviour during the post‐breeding period are relevant when identifying potential energetic bottlenecks, known to be key drivers of seabird population dynamics

Dynamische Analyse von rekonstruierten 3D Punkten zur Navigation eines Stereo-Vision basierten mobilen Systems

Die verschiedenen Anwendungen im Bereich der Navigation von mobilen Systemen hängen u.A. von der Art des verfügbaren Wissens über die Umgebung ab. Anwendungen in Gebäuden verwenden häufig Landmarken oder -karten zur Orientierung oder man verfügt über eine genaue Kenntnis von Objekten, die sich in derartiger Umgebung befinden [2] [6] [9]. Ist nun kein apriori-Wissen über die Umgebung oder Objekte in dieser vorhanden, so sind allgemeine Ansätze zur Navigation gefordert. In diesem Beitrag wird ein Verfahren zur Navigation eines autonomen mobilen Systems in Innenräumen skizziert, das auf rekonstruierten 3D Punkten basiert. Es wird keine Kenntnis über die Umgebung vorausgesetzt. Grundlage dieses Verfahrens ist ein Liniensegment-Stereo-Algorithmus, sowie ein 3D Rekonstruktionsverfahren. Damit erhält man unsichere 3D Punkte einer Szene, die alle als mögliche Hindernisse aufgefaßt werden. Diese 3D Punkte werden mit einem kombinierten statistischen Verfahren analysiert, das schließlich zu einem Lenkwinkel für das mobile System führt. Um die Navigation robuster zu gestalten, wird der in [8] vorgestellte Ansatz um eine dynamische Analyse der rekonstruierten 3D Punkte erweitert. Das Verfahren wurde auf dem Experimentalsystem MOVILAR realisiert (MObile Vision and LAser based Robot), welches über ein Stereokamerasystem verfügt

Automatic Video Analysis for Annotation of Human Body Motion in Humanities Research

The analysis of multi-modal audio-visual data is the very first step to perform research on gestural behaviour in a variety of disciplines such as psycho-linguistic, psychology, behaviour analysis or sign language. The annotation of human gestures and motion of hands is a very time consuming process and requests a lot of effort in terms of personal. Due to the large amount of available data in existing corpora, much video material has not been annotated and even not touched, i.e. valuable material cannot be exploited for research. Thanks to modern video processing algorithms some of the most time consuming annotation tasks can be performed automatically. In this paper, we present a video analysis tool that is specifically designed to perform automatic analysis and annotation of video material for the above mentioned research domains. The presented algorithm provides a large variety of annotations required for behaviour analysis without any user interaction in a fully automatic way. The proposed video analysis tool is currently designed to provide annotations according to the NEUROGES coding system for gestural behaviour, but it can provide also other means of annotations for other coding schemes